Locating Gestures within a Cross-Media Relations Model

Intelligent multimedia systems require cross-media decision mechanisms for integrating pieces of information expressed by different modalities. However, one still strives to find a cross-media relations model that will be general enough to describe how different verbal and non-verbal modalities interact with one another and that will also have computational applicability. In this paper, we explore whether gesture-language interaction could fit into a computational model that we have suggested for modeling image-language interaction. We argue that gestures not only fit in the model, but their unique characteristics —as manifested through their interaction with language— contribute to the model reinforcing its descriptive power and rendering it closer to a cross-media relations one.